HTTP callbacks (aka Webhooks) are great for sending notifications to remote servers in realtime. In most setups, the URLs to contact are provided by foreign entities. All your application needs to do is allow such URLs to be registered, and then hit them whenever interesting things happen. Easy enough, right?

Not so fast. What if someone provides a URL such as http://localhost:10000/destructive-command/ and you’ve got an internal web service running on that port? Under normal circumstances, you might not expect this service to be accessible from the outside. Perhaps you have a firewall, or perhaps the internal service binds explicitly to the localhost interface. Either way, the HTTP callback pattern provides attackers an avenue to access this service from within your internal network, bypassing these kinds of expected security measures.

Possible attacks

Attacks by callback URL are generally limited to making destructive requests as opposed to stealing information, since the attacker does not have a way to receive the response to the request. Most callback invocations use the POST verb, which offers some destructive potential. If you have internal web services that don’t require authentication or special headers to function, then it may be possible to craft callback URLs to invoke methods on these services. Verb overrides via query parameter could make for a lot of fun too. I’m looking at you, POST http://localhost:10000/some/resource/?method=DELETE.

Requests can be made against services on the same machine or other machines on the same network. With lots of network-accessible devices these days, even your printer may not be safe. Granted, the attacker will have constraints on the kinds of requests that can be made in this way, but the fact that requests can be made at all should be alarming.

Mitigation

Here are some ways you can go about protecting your internal network:

  1. Ensure that none of your internal services can be affected by URLs that fit the constraints of your callback scheme.
  2. Require authentication on all internal services.
  3. Run all callbacks through a server that lives on a separate network with no access to any other servers and not running any vulnerable local services of its own.
  4. Have the caller of the URL ensure that it never targets an internal service.

The first option is hard to be confident about. If you’re going to say all current and future internal interfaces will be “safe enough” from evil callback URLs, then you may as well go with the second option for full confidence. The second option feels like overkill though, adding a level of obnoxiousness to internal development.

On the surface, the third option seems to be the most straightforward. However, there’s still the issue of securing the callback server from itself, which sort of takes you back to square one.

I believe the fourth option to be the most ideal. However, it’s harder than it sounds. It’s not enough to simply blacklist certain domains in URLs (e.g. “localhost” or “*.internal” or such), because these domains could still resolve to internal IP addresses. Yes, I can very easily create my own real domain name that resolves to 192.168.1.2, causing it to point to some server on your internal network. What you really need is a blacklist on IP addresses or IP address ranges, that you check after resolving the domain of a URL but before making the HTTP request. Show of hands: how many Webhook implementers are actually doing this?

We’ll discuss how to implement the fourth option below.

IP address blacklisting

The trick to blacklisting URLs by IP address is to extract the domain name from the target URL, resolve it, and then refuse to perform the request if the resolved IP address is on the blacklist. Here’s how to safely hit a URL with Python:

import urlparse
import socket
import httplib

def check_addr(addr):
    if addr.startswith('127.') or addr.startswith('10.'):
        return False
    return True

def call_url_safely(url, data):
    parts = urlparse.urlparse(url)
    host = parts.hostname
    addr = socket.gethostbyname(host)

    if not check_addr(addr):
        raise ValueError('url policy violation')

    port = parts.port
    if port is not None:
        netloc = addr + ':' + str(port)
    else:
        netloc = addr
    conn = httplib.HTTPConnection(netloc)
    uri = parts.path
    if parts.query:
        uri += '?' + parts.query
    conn.request('POST', uri, data, {'Host': host})
    conn.getresponse().read()

call_url_safely('http://localhost:10000/something/', 'somedata')

In the above example, notice that the resolved IP address is passed to the HTTPConnection constructor and that the Host header is explicitly specified when performing the request. This ensures that the HTTP client library connects to the IP address that passed the policy. If we instead went on to make the request against the original domain name, then that could cause the HTTP client library to resolve the domain again and receive a different IP address which might not have passed the policy.

Similar solutions should be possible in other languages/environments, provided your HTTP library allows you to specify an alternative connect host. Note that things can get tricky if you try to do this with HTTPS. You’ll want to ensure your HTTP library can check the server certificate against the original domain and not the IP address.

At Fanout, we send all callbacks through the Zurl HTTP client daemon, which allows configuring policies to be applied on IP addresses & domains prior to making requests. Zurl is also designed to behave correctly with HTTPS. In zurl.conf, we have the lines:

defpolicy=allow
deny=127.*,10.*,192.168.*,*.local

This ensures callbacks can’t go poking around on our internal network. Fanout worker processes never invoke callback URLs directly via HTTP client libraries as this would be unsafe. Other than Zurl, it may be possible to achieve similar centralized protection with an HTTP proxy server such as Squid or Apache, but we have not tried this.

If you’ve implemented Webhooks in your own server application, how have you protected your internal network? We’d love to hear about it.